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ABSTRACT 

An alternative method of measuring the knowledge of students is to use mental models, and specifically concept maps. 
Concept maps provide a visual representation of conceptual and relationship knowledge within a particular domain. 
This study builds on previous studies and attempts to reduce their methodological weaknesses. Students in several 
undergraduate sections of an MIS Telecommunications course were asked to create concept maps of their 
Telecommunications knowledge at three distinct points throughout thesemester. These concept maps were compared 
across all students at a single point in time, within students across the three time periods, and against the concept 
map of a domain “expert” at both the individual and composite levels. Findings indicate that the individual and 
composite maps increased significantly in size over time, and comparisons with the “expert” map show a 
significantly increasing overlap of concepts over time. Moreover, this study shows the applicability of this technique 
as an alternate assessment method. 



INTRODUCTION 

Information systems (IS) educators are tasked with 
preparing students with a broad education in business 
and IS, but measuring students 5 total knowledge can be 
difficult. Throughout a semester, students complete 
quizzes, tests, homework, and other assignments to 
demonstrate their mastery of the particular topic or 
domain of concern. When taken together, these 
measures of knowledge tell how much students know, 
and just as important, what they do not know. Still, 
most of these measures are very structured and limiting 
to the students — they may know 85% percent of the 
course’s material, but the test, quiz, or homework 
assignment may only cover 50% of the material, 
including the 15% they did not know (see Figure 1). 
Moreover, these traditional techniques do not allow the 
students to demonstrate knowledge and mastery beyond 
the assessment technique. To increase knowledge 
measurement accuracy, these traditional assessment 



formats should be supplemented with alternative 
approaches. 

FIGURE 1 

TRADITIONAL ASSESSMENT 
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One alternative approach to knowledge assessment is 
the use of mental models, and specifically concept maps 
or concept webs. When a student creates a visual 
representation of his/her cognitive conceptualization of 
the IS field, viewers of that map get an inside look into 
that student’s mind. The concepts and their 
relationships to each other are represented visually, 
showing the items that the student knows, their 
relationships, and the items that the student does not feel 
are important enough to be included. 

The purpose of this research is to illustrate a potential 
use of concept maps as an assessment tool of students’ 
conceptual knowledge of the IS domain (though 
applicable to other domains), and specifically 
Telecommunications. For the purposes of this research, 
Telecommunications (Telecomm) includes topics such 
as the Internet, networking, cabling, and 
communications, among others, and the regulations that 
govern them. 

RESEARCH QUESTIONS 

Previous studies involving concept maps and the field of 
information systems were limited in their findings (e.g., 
Freeman and Urbaczewski, 2001 ). This was mostly due 
to the methodology and a focus on concept maps drawn 
over a five-week period, and not as a snapshot within a 
short period of time. These studies did not utilize 
concept maps in a traditional assessment scenario, and, 
therefore, a clear insight into the usefulness and merits 
of concept maps has not been provided and there are a 
number of questions left to be answered regarding the 
use of concept maps as assessment techniques. This 
study attempts to answer the following research 
questions. 

1. How does the knowledge of individual students 
change over time? 

2. How does the composite knowledge of students 
change over time? 

3. How does the knowledge of individual students 
compare to that of the “expert” over time? 

4. How does the composite knowledge of students 
compare to that of the “expert” over time? 

The analyses will provide insight into several issues. 
First, the concept maps will show what the students 



know (or at least feel to be important) regarding 
Telecommunications. Second, comparisons of the 
students’ maps to an expert’s map will provide 
information regarding how much is learned from the 
course and whether the concepts that are learned and 
included in the maps are done so “correctly” and as 
intended according to the expert - the faculty instructor. 
Third, comparisons of the three concept maps of 
individual students will provide a measure of how the 
student’s knowledge of Telecommunications changes 
over time. Finally, if there are major differences 
between the student maps and the expert map, changes 
may be necessitated in the teaching of the 
Telecommunications class(es) so that the students have 
closer conceptualizations to the expert. These concept 
maps could potentially be used as an overall assessment 
of the department’s teaching efficacy or simply as a 
view into the students’ minds. 

PRIOR LITERATURE 

The use of mental models, specifically concept maps, 
can aide in the assessment of knowledge at a conceptual 
level (Fisher, 1990; Fisher et al., 1990; Gaines and 
Shaw, 1995; O’Neil and Klein, 1997). Concept maps 
provide a visual representation of conceptual and 
relationship knowledge of main concepts and major sub- 
topics within a particular domain (Hoover and Rabideau, 
1 995). Concept maps consist of nodes that represent the 
concepts and optionally labeled arcs that connect the 
nodes and represent the presence of a relationship. 
Concept maps look like a spider’s web consisting of 
many concepts or nodes connected to each other by lines 
signifying the presence of relationships. Concept maps 
are typically assessed by comparing them to an expert’s 
map in either quantitative or qualitative forms. Other 
assessment techniques include counting the number of 
concepts, the number of relationships, and the number 
of interconnections to determine the degree of 
complexity. See Figure 2 for a sample concept map. 

Other forms of mental models such as cognitive maps, 
semantic networks, and schemata can also be used to 
represent relationships between concepts. However, 
they differ from concept maps in that they also include 
directionality to the relationship or causality between the 
concepts (F isher, 1990). Concept maps were chosen as 
the assessment method because they do not require any 
temporal or cause/effect relationships between the 
concepts. In addition, concept maps support the notion 
of hierarchy, though they do not force a hierarchy. 
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FIGURE 2 

SAMPLE CONCEPT MAP (TABER, 1994) 




A general concept map showing the mapping of 
“concept maps” according to Taber. Note that the links 
are displayed as arrows showing the direction of the link 
(not causation), a convention not always used by others. 

Concept maps have been used for many years as a 
means for communicating knowledge in fields such as 
education, biology, history, mathematics, engineering, 
computer science, and communications (Clibum, 1986; 
Gaines and Shaw, 1995; Wallace andMintzes, 1990; 
Williams, 1995). As seen in Figure 2, concept maps are 
an alternative to linear text, they help reveal 
misconceptions, and they increase learning. Freeman 
and Urbaczewski (1999) describe the use of concept 
maps as an “end-of-[academic] major” assessment 
technique to measure students’ IS knowledge gained 
throughout their coursework. While there may be merit 
in creating and implementing some sort of assessment 
measure to be given to students after they complete all 
of their coursework, this would involve major cultural 
and pedagogical changes and is unlikely to happen at 



most institutions in the near term. Even so, this study 
was an initial attempt at using concept maps as an IS 
educational assessment technique. However, as 
described in Freeman and Urbaczewski (2001), there 
were a number of limitations and weaknesses associated 
with that first study. 

First, there were no comparisons within individuals over 
time. Research suggests that concept maps indeed show 
the differences between novices and experts within a 
field (Markham et al., 1994; Wallace and Mintzes, 
1990), so future studies should analyze the differences 
in concept maps from individuals drawn at different 
points in time. A second weakness was that the 
assignment covered a five-week period, as opposed to 
one or more snapshots in time. Concepts maps drawn 
over five weeks will likely be different from concept 
maps drawn within a time period of several hours, or 
shorter. Third, while the study focused on individual 
students and their individual concept maps, there was no 
control over the students to assure the researchers that 
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there was no outside collaboration by the students 
during the five-week time period. 

Finally, it should be noted that students have typically 
found concept maps to be fun (Freeman and 
Urbaczewski, 1999; Taber, 1994), an emotion unlikely 
to be expressed with regard to many other traditional 
assessment methods like exams and quizzes. 

RESEARCH METHODOLOGY 

This current study takes the composite knowledge of the 
above literature review and, with a focused effort based 
on the concept mapping literature from other disciplines, 
attempts to eliminate the weaknesses of this earlier work 
within IS. 

Eighty-five students in an undergraduate MIS 
telecommunications course at a large U.S. university 
were the subjects for this study. At three different times 
throughout the Spring 2001 semester — during the first 
week of class, during the eighth week of class, and 
during the last week of class — a third party (not the 
instructor) visited the classes and witnessed the students 
as they created concept maps centered around 
Telecommunications. Prior to the first visit, the students 
were given a brief ( 1 5 minutes) introduction and training 
session on the subject of concept maps, how they can be 
drawn, and what they can show the reader. The students 
were given a blank sheet of paper to use in creating their 
concept maps, and they were given 20 minutes to create 
their maps. Students were not permitted to look at 
earlier maps in creating a current map, nor were they 
allowed to keep copies of the maps tfyat they created. 

As an incentive to participate, any student that 
completed all three concept maps during the semester 
received three bonus percentage points towards his/her 
overall course grade. Sixty-two students completed all 
three maps, and those sets of maps are the objects of 
analysis. 

ANALYSIS 

Analysis of the concept maps consisted of the following: 

Coding Scheme 

Each map was redrawn verbatim, but only focusing on 
and including the concepts and relationships regarding 
Telecomm. The cutoff for inclusion was to not include 
any concept and the associated relationships if that 



concept would not be included in a typical textbook 
chapter on telecommunications or internetworking. 
Therefore, “web-enabled database” would be included, 
but its relationship to “SQL” and “SQL” as a concept 
would be beyond the scope of Telecomm. Two outside 
coders, unaffiliated with the study, completed these 
redraws and the rest of the coding process to prevent any 
undue biases on the part of the authors from creeping 
into the analysis. Both authors independently compared 
the redraws for consistency and for concept 
categorization. 

Following these redraws, the maps were randomly split 
into two groups. Using the maps from Group A, a set of 
consistent terms was developed to create a consistent 
coding scheme for all of the maps. In other words, if 
one map contained “employee” and another map 
contained “worker,” or if one map contained “cabling” 
and another map contained “wiring,” the coding scheme 
was necessary to select one term to be used in all 
instances of these synonyms to keep all of the maps 
consistent with each other. This coding scheme was 
then applied to Group B, and the maps from Group B 
were redrawn using the new coding scheme. Any 
concepts that were part of Group B, but not included in 
the coding scheme, were added to create a more robust 
scheme. The maps from Group A were then redrawn 
using this revised coding scheme. This process was 
repeated until all concepts from Groups A and B were 
accounted for in the coding scheme and all of the maps 
had been redrawn using the finalized coding scheme. 
Once the maps were redrawn with the finalized coding 
scheme, each concept and relationship from every map 
was entered into a software package for manipulation 
and analysis. 

General Statistics 

The final coding scheme contained 75 1 unique concepts, 
though there were over 1,600 total concepts (when 
including synonyms) used by the students. The 186 
maps (62 students x 3 maps each) contained 5,000 total 
concepts in 2,606 unique relationship pairs, and over 
5,100 total relationships. Across all maps from all 
rounds, there was a mean of 27 concepts and 28 
relationships on each map. The smallest map had 7 
concepts and the largest had 50. These same two maps 
also anchored the range of relationships with 6 
relationships and 55 relationships, respectively. Of all 
1 86 maps, only three had less than 1 0 concepts, and only 
6 had less than 10 relationships. 
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Individual Maps 

Prior research as well as everyday classroom experience 
indicates that the student concept maps should increase 
in size over time, representing greater domain 
knowledge. Fifty-seven (out of 62) students had larger 
maps in the second round than in the first round. 
Additionally, 49 students had larger maps in the third 
round than in the second round For 48 students, their 
third round map was the largest of the three rounds, and 
for 44 students, each round’s map was larger than the 
previous round. Overall, therefore, the maps of 
individual students did increase in size throughout the 
study. 

Composite Maps 

More details regarding the changes in the individual 
maps over time can be learned from analyses of the 
composite maps from each round. Table 1 presents 
descriptive statistics for each round. 



TABLE 1 

DESCRIPTIVE STATISTICS 
FOR COMPOSITE MAPS 





Round 

1 


Round 

2 


Round 

3 


P- 

value 


Unique Concepts 


380 


418 


459 


n/a 


Total Concepts 


1175 


1742 


2083 


n/a 


Unique 

Relationships 


841 


1133 


1271 


n/a 


Total 

Relationships 


1196 


1789 


2147 


n/a 


Mean Concepts 


18.95 


28.10 


33.60 


<0.001 


Mean 

Relationships 


19.29 


28.89 


34.63 


<0.001 


Complexity 


1.34 


1.79 


2.03 


0.127 



Overall, the maps were significantly larger in each 
subsequent round in terms of concepts and relationships. 
They also increased in complexity in each subsequent 
round (a measure of the number of relationships 
depicted in the map beyond the minimum necessary to 
connect all of the concepts linearly), though not at a 
significant level. 



Based on the mean number of concepts and the mean 
number of relationships in each of the three rounds and 
overall, the following four tables (Table 2-Table 5) 
present the most common concepts and relationships for 
the respective rounds and overall. More concepts and 
relationships are included in all four tables in order to 
accurately represent the frequency distribution and not 
omit concepts or relationships with the same frequency. 
In other words, while Round 1 had a mean of 19 
concepts, the top 22 concepts are shown in Table 2 to 
complete the frequency of all concepts appearing on 12 
maps. Similar adjustments have been made to the 
concepts and relationships in Tables 2-5. These 
“amended means” will also be used below for 
comparisons to the expert’s map. 

The Expert’s Map 

In addition to the student maps, an independently-drawn 
concept map of an “expert”— the faculty member who 
taught these sections of Telecommunications — was 
collected for analysis and comparisons. The expert’s 
map contained 86 conceptsand 1 03 relationships, giving 
a complexity score of 18. For comparison, the largest 
student map contained 50 concepts and 55 relationships 
(also the maximum), and this map was in the third 
round. The maximum complexity score for any of the 
student maps was 10, but this was in the first round. 
Therefore, without any additional comparisons and 
analyses, it is already quite apparent that the expert’s 
map is much larger and more complex than any map 
drawn by the students. 

Even so, comparisons can be made with the individual 
maps against the expert’s map, and with the composite 
maps and summary round data against the expert’s map. 
Every student map had at least one concept in common 
with the expert’s map, and some maps had up to 23 
concepts in common. While these 23 concepts only 
represent 27% of the concepts on the expert’s map, they 
represent 64% of the student’s map. This indicates that 
while nearly every map was much smaller in size than 
the expert’s map, some of the maps contained a 
substantial amount of overlap in terms of the concepts 
included. 

Table 6 shows the number of concepts, on a composite 
level, contained in both the students’ maps and the 
expert’s map. The increasing overlap with the expert’s 
map in each subsequent round was significant for all 
three measurements — actual number of overlapping 
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TABLE 2 

ROUND 1 MOST COMMON CONCEPTS AND RELATIONSHIPS 



Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


63 


Telephone:Telecommunications 


23 


Telephone 


34 


TelecommunicationsComputers 


18 


Internet 


33 


Telecommunications:Networks 


18 


Computers 


32 


Telecommunications:Intemet 


17 


Cellular 


30 


Telecommunications:Business 


12 


Networks 


24 


Telephone:Cellular 


11 


Email 


22 


TelecommunicationsCommunication 


10 


Business 


21 


Telecommunications:Fax 


9 


Cabling 


21 


Telecommunications:Technology 


9 


Individuals 


16 


Telecommunications:Individuals 


9 


LAN 


16 


Wireless:Telecornmunications 


9 


Satellites 


15 


Telecommunications:Cellular 


7 


Wireless 


15 


TelecommunicationsJnformation 


7 


Fax 


13 


Networks:LAN 


7 


Networking 


13 


WAN:Networks 


7 


Video Conferencing 


13 


Telecommunications: Satellites 


7 


WAN 


13 


Video Conferencing:Telecommunications 


6 


AT&T 


12 


TelevisiomTelecommunications 


6 


Modem 


12 


TelecommunicationsiCabling 


6 


Pagers 


12 


Internet [Computers 


6 


Speed 


12 


Telecommunications: Speed 


6 


Technology 


12 


Networks Computers 


6 






Telecommuting:Telecommunications 


6 






TABLE 3 






ROUND 2 MOST COMMON CONCEPTS AND RELATIONSHIPS 




Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


63 


Telephone:Telecommunications 


22 


Telephone 


40 


Wireless:Telecommunications 


19 


Wireless 


35 


TelecommunicationsNetworks 


16 


Fiber Optic 


32 


Transmission Media:Fiber Optic 


13 


LAN 


32 


Transmission Media:Tdecommunications 


13 


Protocols 


27 


TelecommunicationsComputers 


12 


WAN 


27 


TelecommunicationsNetworking 


1 1 


Internet 


26 


Telecommunications:Intemet 


1 1 


TCP/IP 


26 


TelecommunicationsProtocols 


1 1 


Transmission Media 


26 


TCPAP:Protocols 


1 1 


Cellular 


25 


Networks:LAN 


1 1 


Star 


25 


Wireless:Cellular 


1 1 


Network Topology 


24 


Twisted Pair:Transrrrission Media 


10 


Networks 


24 


Telecommunications:LAN 


10 


Cabling 


23 


WAN:Networks 


10 


Twisted Pair 


23 


StanNetwork Topology 


10 


Analog 


20 


TelecommunicationsNetwork Topology 


9 


Data 


20 


Network Topology:Bus 


9 


Email 


20 


Telephone:LATA 


8 


Modem 


20 


Telecommuting:Telecommunications 


8 


Bus 


19 


TelecommunicationsData 


8 


Digital 


19 


Networking^ AN 


7 


Computers 


18 


TelecommunicationsData Sharing 


7 


Ring 


18 


Ring:Network Topology 


7 


LATA 


16 


Wireless:PDAs 


7 
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Table 3 
(continued) 



Concept 


# of Maps 


Relationship 


# of Maps 


Microwave 


16 


WAN :Networking 


6 


Phone Companies 


16 


TCP/IP: Internet 


6 


Coaxial Cable 


15 


TelecommunicationsConnections 


6 


Ethernet 


15 


WAN :T elecommunications 


6 


OSI Model Structure 


15 


TelecommunicationsrTechnology 


6 


Voice 


15 


WWW: Internet 


6 






Transmission Media:Cabling 


6 






TelecommunicationsSecurity 


6 






TelecommunicationsCommunication 


6 






TeIecommunications:Switched Circuit Service 


6 






TelecommunicationsOSI Model Structure 


6 






TelecommunicationsError Checking 


6 



TABLE 4 

ROUND 3 MOST COMMON CONCEPTS AND RELATIONSHIPS 


Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


63 


Wireless:Telecommunications 


25 


Wireless 


51 


Telecommunications.Tntemet 


23 


Telephone 


44 


Wireless:Cellular 


22 


Internet 


43 


Telephone:Telecommunications 


19 


Cellular 


36 


Transmission Media:Fiber Optic 


17 


LAN 


36 


Telecommunications:Networks 


17 


Fiber Optic 


35 


Telecommunications:Security 


17 


DSL 


32 


Security:FirewalI 


14 


WAN 


32 


Twisted Pair:Transmission Media 


14 


Security 


31 


Transmission Media:Tdecommunications 


13 


Bus 


29 


TCP/IP Protocols 


13 


Twisted Pair 


29 


Telecommunications:LAN 


12 


Star 


28 


Star:Network Topology 


12 


Modem 


27 


Network TopoIogy:Bus 


12 


Transmission Media 


27 


WAN Telecommunications 


1 1 


Cabling 


26 


Wireless:PDAs 


10 


Network Topology 


25 


TeIecommunications:Hardware 


10 


Networks 


25 


Networks:Network Topology 


10 


Email 


24 


Network Topology :Mesh 


10 


Ethernet 


23 


Networks :L AN 


10 


Ring 


22 


Ring:Network Topology 


9 


TCP/IP 


21 


Wireless:Satellites 


9 


Digital 


20 


TeIephone:Switched Circuit Service 


9 


Mesh 


20 


WAN:Networks 


9 


Protocols 


20 


TeIecommunications:Switched Circuit Service 


8 


Satellites 


20 


NICs:Hardware 


8 


Video Conferencing 


20 


Wireless:Bluetooth 


8 


Analog 


19 


Video Conferencing:TeIecommunications 


8 


Firewall 


19 


Telecommunications:Network Topology 


7 


Hardware 


19 


TelecommunicationsData Sharing 


7 


Hubs 


19 


TelecommunicationsProtocols 


7 


Router 


17 


TelecommunicationsDSL 


7 


Microwave 


16 


Protocols:IPX/SPX 


7 


Coaxial Cable 


15 


Telecommunications:Cellular 


7 


MAN 


15 


Wireless:Microwave 


7 


Switched Circuit Service 


15 
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TABLE 5 

OVERALL MOST COMMON CONCEPTS AND RELATIONSHIPS 



Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


189 


Telephone:Telecommunications 


64 


Telephone 


1 18 


Wireless:Telecommunications 


53 


Internet 


102 


Telecommunications:Intemet 


51 


Wireless 


101 


Telecommunications:Networks 


51 


Cellular 


91 


Wireless:Cellular 


38 


LAN 


84 


TelecommunicationsComputers 


36 


Fiber Optic 


76 


Transmission Media:Fiber Optic 


31 


Networks 


73 


Networks:LAN 


28 


WAN 


72 


WAN:Networks 


26 


Cabling 


70 


TCP/IP:Protocols 


26 


Email 


66 


Transmission Media:Tdecommunications 


26 


Computers 


63 


Telecommunications:LAN 


25 


Modem 


59 


Telecommunications:Security 


24 


Star 


56 


Twisted Pair:Transmission Media 


24 


Transmission Media 


56 


Star:Network Topology 


22 


Twisted Pair 


55 


TelecommunicationsCommunication 


21 


TCP/IP 


53 


Telephone:Cellular 


21 


DSL 


51 


Telecommunications:Business 


21 


Protocols 


51 


Network Topology.Bus 


21 


Bus 


50 


Telecommunications.Technology 


20 


Network Topology 


50 


Telecommunications:Networking 


19 


Security 


47 


Telecommuting:Teleoommunications 


19 


Digital 


46 


Security:Firewall 


19 


Satellites 


46 


TelecommunicationsProtocols 


19 


Video Conferencing 


46 


Video Conferencing:Telecommunications 


18 


Business 


43 


Telecommunications:Cellular 


18 


Analog 


42 


WANT elecommunications 


18 


Ethernet 


42 


Wireless:PDAs 


18 


Ring 


42 







TABLE 6 

OVERLAP BETWEEN 
INDIVIDUAL AND EXPERT MAPS 





Round 

1 


Round 2 


Round 3 


p-value 


Mean Overlap 
Concepts 


4.68 


10.76 


12.87 


<0.001 


Percent of 

Students’ 

Maps 


24.39% 


37.76% 


38.25% 


<0.001 


Percent of 
Expert’s Map 


5.44% 


12.51% 


14.97% 


<0.001 



concepts, percentage of the students’ maps that overlap 
with the expert’s map, and percentage of the expert’s 
map contained in the students’ maps. 

There are two additional ways to compare the expert’s 
map with the composite maps from each of the three 



rounds. The first is by comparing the most common 
concepts from each round to the concepts used by the 
expert. In Round 1, out of the 22 (“amended mean”) 
most common concepts, only 6 (27.27%) were also a 
part of the expert’s map. For Round 2, this number 
jumps to 21 out of 31 (67.74%). However, the number 
falls to 22 out of 36 (61.11%) for Round 3. Still, this 
does indicate a fair amount of overlap between the 
composite maps from each round and the expert’s map. 
The second comparison is determining the number of 
concepts used by the expert that appear in any of the 
students’ maps in a given round. Of the 86 concepts in 
the expert’s map, 38 did not appear on any students’ 
map in Round 1 , 22 did not appear on any students’ map 
in Round 2, 14 did not appear on any students’ map in 
Round 3, and 6 never appeared on any students’ map in 
any of the rounds, again indicating an overall overlap 
between the students’ maps and the expert’s map, 
especially in the later rounds. 
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Course Grades 

The final analysis is the correlation between the 
students’ final course grades (based on traditional 
assessments) and their concept maps from Round 3. 
The course grades were converted to a GPA based on a 
4.0 being an A. Table 7 shows these correlations. 



TABLE 7 

CORRELATIONS OF CONCEPT MAPS 
WITH COURSE GPA 





GPA 


p-value 


Concepts 


0.376 


0.003 


Relationships 


0.370 


0.003 


Complexity 


0.044 


0.732 



The positive and significant correlations between GPA 
and both Concepts and Relationships indicate that the 
students who performed well in the traditional 
assessments also produced concept maps that were 
larger with more relationships. While the correlation 
with GPA and Complexity was slightly positive, it was 
not significant as expected based on the relatively low 
complexity of the concept maps. 

DISCUSSION 

The results are fairly clear: as the students progressed 
throughout the semester, their concept maps 
significantly increased both in size and in similarity to 
the expert’s map. We would expect any other 
assessment method to yield similar results, hopefully as 
a measure of learning. 

One might wonder if the increasing number of concepts 
and relationships would be a self-fulfilling prophecy. 
That is to say, that certainly by the end of the semester, 
the students would have been exposed to more material 
and would have been able to express more of it on their 
maps. This is probably true to some extent. Of 
importance to us, additionally, is theirability to continue 
to recall basic topics that were primarily discussed in the 
beginning of the semester, as well as relate those topics 
to their understanding of Telecommunications before 
beginning the course. This demonstrates additive 
knowledge on the reshaping of prior knowledge. 

If we look at the relationship between the average 
overlap of concepts, even at its highest (12.87 concepts: 
38.25% of the students’ maps, 14.97% of the expert’s 
map), we may conclude that the transfer of knowledge 



is limited and that both the professor and students 
should be given failing grades. Is this so? We do not 
believe it is. We believe that this is rather a celebration 
of the educational system and the abilities of individuals 
to focus in areas that may not be of primary interest to 
the expert. This is the ability to inspire. Of course there 
must be some base level of knowledge in order for one 
to inspire, and we at least find that in the increasing 
number of concepts represented in the students’ maps. 
But to suggest that the goal is to have the students 
repeating the course material as a mantra upon 
completion would imply that our universities are no 
more than memorization camps. These maps are one 
more tool in getting inside the minds of our students and 
observing the relative importance of certain concepts. 

In the results, we find that course grade is positively and 
significantly correlated with both number of concepts 
and number of relationships in the individual maps. 
This can be attributable to a number of factors. First of 
all, and what we as educators are more likely to espouse, 
it is a demonstration of understanding of the subject 
matter. While it is true that the individual who does not 
know anything about the subject could not draw a 
concept map similar to an expert’s map, the opposite 
relationship is not necessarily true. We also cannot say 
for sure that the primary cause of the correlation is 
increased knowledge. Other mitigating factors, such as 
motivation, attitude towards course, attitude towards 
instructor, or attitude towards the researcher could all 
play a role in identifying the reasoning behind the 
correlation. For the purposes of this paper, we make no 
claim beyond the existence of positive and significant 
correlations. Future research should attempt to measure 
these differences. 

We find no relationship between grade in course and 
complexity. This is likely related to the low overall 
complexity of all studentmaps. Future research should 
also investigate the reasoning behind relatively low 
complexity and the mental model of the student. For 
example, does the student view knowledge as largely 
hierarchical, or as an interconnected web of concepts, 
and how does the student demonstrate that knowledge? 

These concept maps allow the instructor to “check” on 
what students understand and know regarding the course 
and his/her expectations. Of course, quizzes and exams 
attempt to do just that, but they are limited in what can 
be conveyed and in what can be covered. Additionally, 
beyond the quantitative analyses that appear in this 
paper, there are several valuable qualitative analyses 
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that can be performed on the concept maps. By looking 
for “strange” relationships within the concept maps, the 
instructor can clearly see what the students don’t 
understand, or at least misunderstood at one point. For 
example, if the relationships “Network Topology:LAN” 
and “Network Topology:Client Server” appeared, it 
would be apparent that there is a misunderstanding of 
Network Topology within the class. These “strange” 
relationships are particularly helpful when uncovered in 
the middle of the course. In the beginning of the course, 
many students will likely have misunderstandings about 
concepts and relationships merely due to the fact that 
they have not yet been covered adequately, if at all. At 
the end of the course, the instructor may not have the 
time to properly correct the misunderstandings before 
the students take part in traditional, graded 
examinations. However, inthe middle of the course, the 
instructor has not only covered a fair amount of the 
material, but also has the necessary time to clarify 
misunderstandings. 

A second qualitative analysis involves the actual 
knowledge and understanding conveyed in the concept 
maps. Ideally, as the course progresses, the concept 
maps will contain concepts and relationships that 
indicate a deeper understanding of the material. These 
concepts and relationships will go beyond the basic 
terminology of the course, and the relationships will 
become more complex. These qualitative analyses go 
beyond traditional assessment techniques in providing 
the instructor with a much clearer view of what his/her 
students know, think, and understand. 

A potential limitation with interpreting the data is that 
the researchers and coders were required to take the 
topics as they were printed on the paper. It is difficult 
from the writing to ask “is that what he/she really 
meant?” From the context and the relationships, it is 
easy to tell that in one sense “cable” may refer to a type 
of Internet access through a cable TV company, while in 
another sense “cable” may refer to a length of copper 
surrounded by insulation. However, this interpretation 
is certainly not perfect. The alternative, though, would 
be to have students pick words and definitions from an 
approved list. This would certainly defeat the purpose 
of this free association task and almost “force” the 
results. 



CONCLUSION 

This study is an attempt to better assess students’ 
knowledge using concept maps, and an attempt to 
modify previous studies and eliminate their weaknesses. 
Several issues should be noted: the contributions of this 
study and the need for additional research. First, this 
study makes several contributions to research and to 
teaching. It informs the field of IS academics in 
general, and specifically Telecomm academics, of an 
additional method for the assessment of students that 
has gained support in other academic fields such as 
education and psychology. Granted, this method may 
not be appropriate to replace other assessment methods, 
but that will depend on the specific situation and the 
type of knowledge to be assessed. This empirical study 
also demonstrates the applicability of this assessment 
method in a real classroom situation and shows what can 
be learned. 

While the concept maps of these students were 
compared to an expert, this expert was a single faculty 
member and not all of the faculty members that teach or 
have taught the Telecommunications class at this 
particular institution. Student maps could be compared 
to a “departmental” or “institutional” map or even to the 
map at a discipline level from the IS 1997 or IS 2002 
requirements (available through the AIS website, http:// 
aisnet.org/Curriculum/). Future research should also 
analyze concept maps created by groups as an 
alternative method for conveying knowledge. Another 
area for future research is the use of concept maps in 
other ways within the classroom, such as for organizing 
lectures and units, for brainstorming, or for bringing 
together diverse viewpoints. 

The optimal means for assessing student knowledge 
may often seem like the quest for the Holy Grail. While 
concept mapping may not be the “silver bullet” that is 
all-encompassing and useful in every situation, it is one 
more “fly” in the teacher’s “tackle box,” and one that 
should not be ignored. This study attempts to show the 
usefulness of this tool in one situation, and its 
applicability to many other situations can be surmised 
from this study. 
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